Papers by Mohammad Taher Pilehvar
Blind Men and the Elephant: Diverse Perspectives on Gender Stereotypes in Benchmark Datasets (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing benchmarks for measuring gender stereotypical bias in language models are inconsistencies . lack of explicit standards in data gathering can have detrimental effects on results . |
| Approach: | They propose that currently available benchmarks capture only partial facets of gender stereotypes . they apply a framework from social psychology to balance data across components of gender stereotypes based on stereotypical benchmarks. |
| Outcome: | The proposed framework improves correlation between different benchmarks by using simple balancing techniques. |
How Does Fine-tuning Affect the Geometry of Embedding Space: A Case Study on Isotropy (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for fine-tuning pre-trained language models are ineffective, despite their potential, pre-training models suffer from important weaknesses. |
| Approach: | They analyze the extent to which the isotropy of the embedding space changes after fine-tuning. |
| Outcome: | The proposed model improves the isotropy of embedding space after fine-tuning . the model can encode linguistic properties, but lacks the social bias needed to improve it . |
XL-WiC: A Multilingual Benchmark for Evaluating Semantic Contextualization (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing evaluation benchmarks for assessing distinct meanings of words are tied to sense inventories, restricting their usage to knowledge-based representation techniques. |
| Approach: | They propose a multilingual benchmark that models distinct meanings of words in English . they use a binary disambiguation task with gold standards in 12 new languages . |
| Outcome: | The proposed model can model distinct meanings of words in English even when no tagged instances are available for a target language. |
The interplay between lexical resources and Natural Language Processing (N18-6)
Copied to clipboard
| Challenge: | linguistic, world and common sense knowledge is an important research area, but processing and storing it in lexical resources is not a straightforward task. |
| Approach: | They propose to use NLP methods to help process of constructing and enriching lexical resources and the use of lexicals for improving NLP applications. |
| Outcome: | The proposed approach aims to speed up and/or ease up the process of resource curation and enrichment. |
PerCul: A Story-Driven Cultural Evaluation of LLMs in Persian (2025.naacl-long)
Copied to clipboard
Erfan Moosavi Monazzah, Vahid Rahimzadeh, Yadollah Yaghoobzadeh, Azadeh Shakery, Mohammad Taher Pilehvar
| Challenge: | Large language models predominantly reflect Western cultures due to the dominance of English-centric training data. |
| Approach: | They propose a dataset to assess the sensitivity of LLMs to Persian culture. |
| Outcome: | The proposed model shows a 11.3% gap between best closed-source model and layperson baseline while the gap increases to 21.3% by using the best open-weight model. |
Pun Unintended: LLMs and the Illusion of Humor Understanding (2025.emnlp-main)
Copied to clipboard
Alessandro Zangari, Matteo Marcuzzo, Andrea Albarelli, Mohammad Taher Pilehvar, Jose Camacho-Collados
| Challenge: | Existing models for pun detection lack nuanced grasp typical of human interpretation. |
| Approach: | They analyze existing pun detection benchmarks and human evaluation across recent LLMs to find subtle changes in puns that mislead LLM. |
| Outcome: | The proposed models lack the nuance typical of human interpretation and lack the depth of their analysis to detect puns. |
Looking at the Overlooked: An Analysis on the Word-Overlap Bias in Natural Language Inference (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for debiasing are ineffective in addressing the reverse word-overlap bias. |
| Approach: | They propose to investigate the reverse word-overlap bias in NLI models . they find that existing debiasing methods are generally ineffective . |
| Outcome: | The proposed model is biased towards the non-entailment label on instances with low overlap . the proposed model does not have minority examples, the authors show . |
NormXLogit: The Head-on-Top Never Lies (2025.emnlp-main)
Copied to clipboard
| Challenge: | NormXLogit is a novel approach for assessing the significance of input tokens based on word embeddings . |
| Approach: | They propose a novel method for assessing the significance of input tokens based on the input and output representations associated with each token. |
| Outcome: | The proposed method outperforms gradient-based methods in faithfulness and offers competitive performance compared to leading architecture-specific techniques. |
Card-660: Cambridge Rare Word Dataset - a Reliable Benchmark for Infrequent Word Representation Models (D18-1)
Copied to clipboard
| Challenge: | Existing benchmarks for rare word representation are lacking for evaluation and comparison . a task-based evaluation does not provide a solid basis for comparing different models . |
| Approach: | They propose to use an expert-annotated word similarity dataset to evaluate rare word representation techniques. |
| Outcome: | The proposed dataset provides a reliable benchmark for rare word representation techniques. |
FarExStance: Explainable Stance Detection for Farsi (2025.coling-main)
Copied to clipboard
Majid Zarharan, Maryam Hashemi, Malika Behroozrazegh, Sauleh Eetemadi, Mohammad Taher Pilehvar, Jennifer Foster
| Challenge: | FarExStance is a new dataset for explainable stance detection in Farsi . it contains extractive explanations as evidence for stance labels and claims . |
| Approach: | They propose a dataset for explainable stance detection in Farsi with extractive explanations as evidence. |
| Outcome: | The proposed model is the most accurate on stance detection, while the best explanation is from few-shot Claude-3.5-Sonnet. |
An Isotropy Analysis in the Multilingual BERT Embedding Space (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing studies have explored the advantages of multilingual pre-trained models in capturing shared linguistic knowledge. |
| Approach: | They investigate the anisotropic embedding space and outlier dimensions of the multilingual BERT model for two known issues of the monolingual models. |
| Outcome: | The proposed model has no outlier dimension and has highly anisotropic space . the results show that increasing the isotropy of multilingual space can improve its representation power and performance, similar to what had been observed for monolingual CWRs on semantic similarity tasks. |
RepMatch: Quantifying Cross-Instance Similarities in Representation Space (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in dataset analysis have enabled more sophisticated approaches to analyzing and characterizing training data instances. |
| Approach: | They propose a method that characterizes data through the lens of similarity. |
| Outcome: | The proposed method can compare datasets, identify more representative subsets, and uncover heuristics underlying the construction of some challenge datasets. |
Understanding LLM Performance Degradation in Multi-Instance Processing: The Roles of Instance Count and Context Length (2026.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are used for processing multiple documents or analysis over a number of instances. |
| Approach: | They perform a comprehensive evaluation of the multi-instance processing ability of LLMs for tasks in which they excel individually. |
| Outcome: | The proposed model performs well on tasks in which it excels individually. |
DecompX: Explaining Transformers Decisions by Propagating Token Decomposition (2023.acl-long)
Copied to clipboard
| Challenge: | Existing vector-based explanation methods for Transformer-based models are limited in their ability to explain the decisions of multiple layers. |
| Approach: | They propose a vector-based explanation method based on the construction of decomposed token representations and their successive propagation throughout the model without mixing them in between layers. |
| Outcome: | The proposed method outperforms existing vector-based and gradient-based methods on transformer-based models by a wide margin. |
TruthTrap: A Bilingual Benchmark for Evaluating Factually Correct Yet Misleading Information in Question Answering (2026.findings-eacl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are increasingly used to answer factual, information-seeking questions (ISQs). |
| Approach: | They propose to use a dataset to evaluate large language models to generate human-like text on ISQs in two languages, English and Farsi, and then use it to evaluate nine LLMs. |
| Outcome: | The proposed dataset shows that accuracy drops by 25% when models encounter misleading yet factual hints. |
On the Importance of the Kullback-Leibler Divergence Term in Variational Autoencoders for Text Generation (D19-56)
Copied to clipboard
| Challenge: | Variational Autoencoders suffer from learning uninformative latent representations due to issues such as approximated posterior collapse or entanglement of the latent space. |
| Approach: | They propose to impose an explicit constraint on the Kullback-Leibler divergence term inside the VAE objective function to understand the significance of the KL term in controlling the information transmitted through the VAe channel. |
| Outcome: | The proposed constraint avoids posterior collapse, but it also controls the information transmitted through the VAE channel. |
Exploiting Language Model Prompts Using Similarity Measures: A Case Study on the Word-in-Context Task (2022.acl-short)
Copied to clipboard
| Challenge: | Existing few-shot approaches fail on the semantic distinction task of the Word-in-Context dataset. |
| Approach: | They propose a prompt-based approach which boosts few-shot performance to the level of fully supervised methods by using similarity metrics. |
| Outcome: | The proposed technique boosts few-shot performance to the level of fully supervised methods. |
Exploring the Role of BERT Token Representations to Explain Sentence Probing Results (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have focused on enhancing existing models with the primary objective of improving downstream performance on various NLP tasks. |
| Approach: | They propose to use BERT to encode meaningful knowledge in token representations to explain probing results. |
| Outcome: | The proposed model can detect syntactic and semantic abnormalities and distinguish between grammatical number and tense subspaces. |
Embeddings in Natural Language Processing (2020.coling-tutorials)
Copied to clipboard
| Challenge: | Embeddings have been a key topic of interest in NLP for the past decade . a quick warm-up introduction to NLP and why it is important to have a semantic comprehension of texts . |
| Approach: | This tutorial will provide a high-level synthesis of the main embedding techniques in NLP . it will start with word embedds and then move to other types of embeddable vectors . |
| Outcome: | This tutorial will provide a high-level synthesis of the main embedding techniques in NLP . it will start with word embedds and move to other types of embeddable representations . |
DadmaTools: Natural Language Processing Toolkit for Persian Language (2022.naacl-demo)
Copied to clipboard
| Challenge: | Existing tools for Persian language processing are based on conventional non-neural models and do not take full advantage of the latest developments. |
| Approach: | They propose to use a Python neural pipeline for Persian text processing tasks . they use 'parsBERT' to fine-tune the Python pipeline using the PerDT dataset . |
| Outcome: | The proposed toolkit can achieve state-of-the-art performance on multiple NLP tasks. |
On the Importance of Distinguishing Word Meaning Representations: A Case Study on Reverse Dictionary Mapping (N19-1)
Copied to clipboard
| Challenge: | Sense representations target meaning conflation deficiency but their potential impact has not been investigated in downstream NLP applications. |
| Approach: | They propose to use a reverse dictionary system to address meaning conflation deficiency . they propose to integrate senses into the system to improve semantic understanding . |
| Outcome: | The proposed approach can improve the performance of a downstream NLP application. |
Stochastic Fine-Tuning of Language Models Using Masked Gradients (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are the dominant paradigm in Natural Language Processing but fine-tuning them for specific downstream tasks often requires updating a vast number of parameters. |
| Approach: | They propose a method that selectively updates a small subset of parameters in each step of the tuning process. |
| Outcome: | The proposed approach outperforms existing fine-tuning methods while updating merely **0.08**% of the model’s parameters. |
WiC-TSV: An Evaluation Benchmark for Target Sense Verification of Words in Context (2021.eacl-main)
Copied to clipboard
| Challenge: | Existing benchmarks for Word Sense Disambiguation are limited to those systems in which sense distinctions are defined according to an underlying sense inventory. |
| Approach: | They propose a framework for Target Sense Verification of Words in Context which grounds its uniqueness as binary classification task and independent of external sense inventories. |
| Outcome: | The proposed framework is highly flexible for evaluation of diverse models and systems in and across domains. |
Large-scale Exploration of Neural Relation Classification Architectures (D18-1)
Copied to clipboard
| Challenge: | Existing studies on relation classification have been limited to a very narrow range of datasets, making comparisons between approaches difficult. |
| Approach: | They propose a multi-channel LSTM model combined with a CNN that takes advantage of all currently popular linguistic and architectural features. |
| Outcome: | The proposed model achieves state-of-the-art on two datasets and provides direct insights into the challenges faced by language models on relation classification. |
Mapping Text to Knowledge Graph Entities using Multi-Sense LSTMs (D18-1)
Copied to clipboard
| Challenge: | a paper addresses the problem of mapping natural language text to knowledge base entities. |
| Approach: | They propose a model for mapping natural language text to knowledge base entities using a multi-dimensional entity space obtained from a knowledge graph. |
| Outcome: | The proposed model is applied to large-scale text-to-entity mapping and entity classification tasks with state-of-the-art results. |
ParsFEVER: a Dataset for Farsi Fact Extraction and Verification (2021.starsem-1)
Copied to clipboard
Majid Zarharan, Mahsa Ghaderan, Amin Pourdabiri, Zahra Sayedi, Behrouz Minaei-Bidgoli, Sauleh Eetemadi, Mohammad Taher Pilehvar
| Challenge: | Existing methods for fact-checking and verification require large amounts of annotated data, but this is limited to low-resource languages. |
| Approach: | They present a first publicly available Farsi dataset for fact extraction and verification . they use the construction procedure of the standard English dataset for the task . |
| Outcome: | The proposed dataset improves on the standard English dataset and is available on github. |
Evaluating Cultural Knowledge and Reasoning in LLMs Through Persian Allusions (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Allusion recognition is a critical test of LLMs' ability to deploy stored information in open-ended, figurative settings. |
| Approach: | They propose a framework for evaluating Persian literary allusions through annotations and LLM-generated texts incorporating allusion in novel contexts. |
| Outcome: | The proposed framework evaluates Persian literary allusions through annotations and LLM-generated texts incorporating allusion in novel contexts. |
Don’t Discard All the Biased Instances: Investigating a Core Assumption in Dataset Bias Mitigation Techniques (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing techniques for mitigating dataset bias often leverage a biased model to identify biased instances. Existing methods for mitiging dataset bias use shallow patterns that can be exploited by the model. |
| Approach: | They propose to use partial-input and limited-capacity models to detect biased instances and reduce their role during training to enhance its robustness to out-of-distribution data. |
| Outcome: | The proposed method outperforms existing methods for mitigating dataset bias on two well-known datasets in the domain, MNLI and FEVER. |
WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representations (N19-1)
Copied to clipboard
| Challenge: | Existing word embeddings cannot model the dynamic nature of words’ semantics, i.e., the property of words to correspond to potentially different meanings. |
| Approach: | They propose a large-scale Word in Context dataset, called WiC, which is curated by experts and can be used to evaluate context-sensitive representations. |
| Outcome: | The proposed models outperform the standard evaluation dataset for the purpose and highlight their shortcomings. |
Morables: A Benchmark for Assessing Abstract Moral Reasoning in LLMs with Fables (2025.emnlp-main)
Copied to clipboard
Matteo Marcuzzo, Alessandro Zangari, Andrea Albarelli, Jose Camacho-Collados, Mohammad Taher Pilehvar
| Challenge: | Literature-based benchmarks provide a compelling framework for evaluating LLMs' capacity for complex abstract reasoning and inference. |
| Approach: | They propose a novel moral reasoning benchmark built from fables and short stories that uses adversarial variants to stress-test model robustness. |
| Outcome: | The proposed model outperforms models on fables and short stories, but is susceptible to adversarial manipulation and rely on superficial patterns rather than true moral reasoning. |
An Empirical Study on the Transferability of Transformer Modules in Parameter-efficient Fine-tuning (2022.emnlp-main)
Copied to clipboard
| Challenge: | Parameter-efficient fine-tuning is a computationally expensive process . introducing new parameters to an already-large model can be considered a drawback. |
| Approach: | They investigate the capability of different transformer modules in transferring knowledge from a pre-trained model to a downstream task. |
| Outcome: | The proposed methods show that each transformer module is a winning ticket . they show that with only 0.003% updateable parameters, they can show acceptable performance on target tasks. |
Will-They-Won’t-They: A Very Large Dataset for Stance Detection on Twitter (2020.acl-main)
Copied to clipboard
Costanza Conforti, Jakob Berndt, Mohammad Taher Pilehvar, Chryssi Giannitsarou, Flavio Toxvaerd, Nigel Collier
| Challenge: | stance detection is a key component of fake news detection, fact-checking and rumor verification. |
| Approach: | They propose to use a large dataset of English tweets for stance detection for a rumor verification task. |
| Outcome: | The proposed dataset contains 51,284 tweets in English, making it the largest available dataset of the type. |
GlobEnc: Quantifying Global Token Attribution by Incorporating the Whole Encoder Layer in Transformers (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing methods for interpreting the underlying dynamics of Transformers have been criticized for their lack of reliability. |
| Approach: | They propose a token attribution analysis method that incorporates all components in the encoder block and aggregates this across layers. |
| Outcome: | The proposed method significantly outperforms existing methods on saliency scores and correlation with gradient-based salience scores. |
AdapLeR: Speeding up Inference by Adaptive Length Reduction (2022.acl-long)
Copied to clipboard
| Challenge: | Pre-trained language models have shown stellar performance in downstream tasks, but their excessive computational costs and high latency hinder their usage in resource-limited settings. |
| Approach: | They propose a method that dynamically eliminates less contributing tokens through layers, resulting in shorter lengths and consequently lower computational cost. |
| Outcome: | The proposed method shows speedups up to 22x during inference time without much sacrifice in performance. |
Which Melbourne? Augmenting Geocoding with Maps (P18-1)
Copied to clipboard
| Challenge: | Existing methods to associate geographic information in text with coordinates are limited by lexical features and cartesian coordinates. |
| Approach: | They propose a geocoder that exploits implicit lexical clues to associate coordinates with text . they propose encoding of geographic metadata to generate two distinct views of the same text. |
| Outcome: | The proposed method improves state-of-the-art results on three datasets and an open-source dataset for disease outbreaks and epidemics. |
TweetTER: A Benchmark for Target Entity Retrieval on Twitter without Knowledge Bases (2024.lrec-main)
Copied to clipboard
| Challenge: | Entity linking is a well-established task in NLP consisting of associating entity mentions with entries in a knowledge base. |
| Approach: | They propose a benchmark that reframes entity linking as a binary entity retrieval task and uses a knowledge base to evaluate model performance. |
| Outcome: | The proposed benchmark aims to bridge the challenges in entity linking in noisy domains such as social media. |
Guide the Learner: Controlling Product of Experts Debiasing Method Based on Token Attribution Similarities (2023.eacl-main)
Copied to clipboard
| Challenge: | Several proposals have been put forward for improving out-of-distribution performance by mitigating dataset biases. |
| Approach: | They propose a fine-tuning strategy that incorporates the similarity between the main and biased model attribution scores in a Product of Experts (PoE) loss function to further improve OOD performance. |
| Outcome: | The proposed method improves OOD performance while maintaining in-distribution performance. |
A Cluster-based Approach for Improving Isotropy in Contextual Embedding Space (2021.acl-short)
Copied to clipboard
| Challenge: | Existing approaches to address the representation degeneration problem in contextual embedding spaces require a learning process to retrain models with additional objectives. |
| Approach: | They propose a local cluster-based method to address the representation degeneration problem in contextual embedding spaces by removing local dominant directions from verb representations. |
| Outcome: | The proposed method improves CWRs performance on semantic tasks by removing dominant directions of verb representations. |
On the Importance of Data Size in Probing Fine-tuned Models (2022.findings-acl)
Copied to clipboard
| Challenge: | Several studies have investigated the reasons behind the effectiveness of fine-tuning, usually through the lens of probing. |
| Approach: | They propose to investigate the reasons behind the effectiveness of fine-tuning by examining the impact of data size on the extent of encoded linguistic knowledge. |
| Outcome: | The proposed probes show that the size of the training data affects the recoverability of the changes made to the model’s linguistic knowledge. |
STANDER: An Expert-Annotated Dataset for News Stance Detection and Evidence Retrieval (2020.findings-emnlp)
Copied to clipboard
Costanza Conforti, Jakob Berndt, Mohammad Taher Pilehvar, Chryssi Giannitsarou, Flavio Toxvaerd, Nigel Collier
| Challenge: | a new news dataset targets both stance detection (SD) and fine-grained evidence retrieval (ER) . stance Detection (SD), which is a form of multitask learning, has gained increasing interest in recent work . |
| Approach: | They propose a news dataset that targets both stance detection (SD) and fine-grained evidence retrieval (ER) their dataset is an expert-annotated news dataset with 3,291 articles. |
| Outcome: | The proposed dataset is a high-quality benchmark for future research in stance detection and evidence retrieval. |
Generating Knowledge Graph Paths from Textual Definitions using Sequence-to-Sequence Models (N19-1)
Copied to clipboard
| Challenge: | a novel method for mapping unrestricted text to knowledge graph entities is proposed . a proof-of-concept experiment has encouraging results comparable to those of state-of the-art systems. |
| Approach: | They propose a method for mapping unrestricted text to knowledge graph entities by framing the task as a sequence-to-sequence problem. |
| Outcome: | The proposed method produces highly interpretable predictions comparable to state-of-the-art systems. |
Incorporating Stock Market Signals for Twitter Stance Detection (2022.acl-long)
Copied to clipboard
Costanza Conforti, Jakob Berndt, Mohammad Taher Pilehvar, Chryssi Giannitsarou, Flavio Toxvaerd, Nigel Collier
| Challenge: | stance detection is the task of automatically classifying the writer's opinion expressed in a text towards a particular target. |
| Approach: | They propose a robust multi-task neural architecture that combines textual input with high-frequency intra-day time series from stock market prices. |
| Outcome: | The proposed system achieves state-of-the-art on the wt–wt dataset. |
Synthia: Scalable Grounded Persona Generation from Social Media Data (2026.acl-long)
Copied to clipboard
| Challenge: | Persona-driven large language models (LLMs) are increasingly used in computational social science, yet their validity critically depends on the fidelity of the underlying personas. |
| Approach: | They propose a persona-generation framework that grounds LLM-generated personas in real social-media posts while delegating narrative construction to language models. |
| Outcome: | The proposed framework outperforms state-of-the-art methods for most demographics across different dimensions while maintaining interaction graph structure among personas grounded in real social network users. |